Summary

When a SLURM job is submitted it may sit in the pending queue for minutes or hours with no feedback. This PR adds a lightweight background daemon thread that polls squeue --start every 30 seconds and prints the estimated start time to stdout, stopping automatically once the job leaves the pending queue.

• _poll_job_start_time: new method on SlurmTunnelScheduler that runs squeue --start --noheader -j {job_id} -o '%i|%S|%T' in a loop, printing [SLURM] Job {id} - State: PENDING, Estimated start: <time> until the job starts or the stop event is set
• schedule(): starts a daemon thread after _save_job_dir; stops any pre-existing thread for the same job_id first (retry/duplicate case)
• _cancel_existing(): signals the polling thread when a job is cancelled
• close(): replaces the ... stub — signals all polling threads and clears the tracking dicts

Edge cases handled:

• stdout=None guard (result.stdout or "")
• Non-zero return_code treated as empty (SLURM error text in stdout)
• Array jobs (12345_1, 12345_2) deduplicated — only first line printed per cycle
• Exception resilience — logs debug and retries after 30s wait
• Duplicate job_id on retry — old thread is stopped before new one starts
• Old SLURM without --me flag — uses -j {job_id} only

Test plan

• 11 new TDD tests covering all edge cases above (test/run/torchx_backend/schedulers/test_slurm.py)
• 2 existing test_schedule* tests updated to patch _poll_job_start_time (prevents polling thread from interfering with tunnel.run.assert_called_once())
• All 33 tests in the file pass: uv run -- pytest test/run/torchx_backend/schedulers/test_slurm.py -v
• ruff check + ruff format clean

🤖 Generated with Claude Code